Mv level work3 ... change from 3 overlapped levels to 2 #84
Conversation
…vels from 3 (upgrade issue). adjust VersionSet::Finalize() penalty values again based upon testing.
…leared via compactions (Riak 1.3 to 1.4 conversion). Also cleaned some int versus size_t compiler warnings.
…inalize() back to 8, was 2 recently (2i_slf_stressmix test)
…umber of times a compaction thread call PrioritizeWork() to reduce unnecessary overhead. compactions are running too slow in 2i_slf_stressmix
ghost
assigned 
|
I have not been able to find any issues with the code beyond that CorruptionTest.TableFileIndexData test failure. It would be great to fix that so the test suite passes cleanly. Don't forget to create an issue to test a change to handle the throttle skipping path in DBImpl::Write. With those caveats, 👍 |
…processing of recovery logs properly post to version_set before starting background compactions.
|
In addition to looking at the code, ended up testing the code in a few scenarios. One scenario was writing data using LevelDB from Riak 1.3.2 and then upgrading the node to this branch of LevelDB. The goal was to trigger the overlap compaction logic. In preliminary testing earlier today, I was able to hit this case but then ran into repeated errors in the LOG of the form: This issue has successfully been resolved in the latest commit 206b277. Specifically, despite being able to repeatedly reproduce the issue prior to this commit, I cannot reproduce it after. Furthermore, the logic in the commit makes complete sense to me: ie. delaying the forced compaction logic until after the recovery code has run. Also, since the call to One thought that crossed my mind would be to add something to the LOG output that notes the number of overlapping levels being used, similar to how to print out the write buffer / bloom filter version / etc to the LOG. The more information we have in the LOGs the better. In any case, this is just a suggestion and is future work at that. Note for historical purposes: issue #85 arose out of internal discussion on this pull-request. It is future work to keep the code change here minimal and move along with current release plans. +1 merge |
Details here: https://github.com/basho/leveldb/wiki/Mv-level-work3
Three overlapped levels gives us better performance, especially in write heavy and in read recent heavy environments. However, the conversion time to shift 1.3 level files into 1.4 formation would have taken hours on big installations. So only doing one extra overlapped this release. It will also smooth the way for sorter upgrade from 2 to 3 in a future release.